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Article Info ABSTRACT 

Article history: Sentiment analysis or opinion mining is used to understand the community’s 
opinions on a particular product. This is a system of selection and 

Received Oct 25, 2021 classification of opinions on sentences or documents. At a more detailed 

Revised Feb 9, 2022 level, aspect-based sentiment analysis makes an effort to extract and 

Accepted Feb 27, 2022 categorize sentiments on aspects of entities in opinion text. In this paper, we 


propose a novel supervised learning approach using deep learning 
techniques for a multitasking aspect-based opinion mining system that 
Keywords: supports four main subtasks: extract opinion target, classify aspect, classify 
entity (category) and estimate opinion polarity (positive, neutral, negative) 
on each extracted aspect of the entity. We have used a part-of-speech (POS) 


Aspect-based sentiment 


multitask layer to define the words’ morphological features integrated with GloVe 
Bidirectional long-short term word embedding in the previous layer and fed to the convolutional neural 
memory network_bidirectional long-short term memory (CNN_BiLSTM) stacked 
Convolutional neural network construction to improve the model’s accuracy in the opinion classification 
Part-of-speech tag process and related tasks. Our multitasking aspect-based sentiment analysis 
Word embedding experiments on the dataset of SemEval 2016 showed that our proposed 
models have obtained and categorized core tasks mentioned above 
simultaneously and attained considerably better accurateness than the 
advanced researches. 
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1. INTRODUCTION 

Collecting customer feedback is a great way for businesses to understand the strengths and 
weaknesses of their products and services and quickly grasp the psychology and needs of customers to bring 
them the most perfect products and services. This collection requires an automatic opinion understanding, 
exploration, and analysis system. Traditional opinion mining systems mainly focus on identifying opinions in 
sentences or entire documents. This leads to a drawback: the defined sentiment polarity is general to the 
whole text. However, in some cases, users will express their personal views on each aspect of the entity 
mentioned rather than stating their opinions on the whole text. Aspect-based sentiment analysis (ABSA) with 
the target to categorize opinion represented on each extracted aspect of the entity in opinion text is an 
efficient resolution for this trouble. Most recent sentiment analysis researches have focused on one of two 
main approaches [1]: (4) based on lexicon and (ii) based on machine learning. 
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In (i) measures are used to estimate the text’s sentiment orientation based on the words’ sentiment 
tendency identified on the degree of correlation between opinion words in the opinion corpus and the data. 
Some of the studies illustrate this approach. Khan et al. [2], [3] applied WordNet and SentiWordNet to 
calculate semantic scores of words in the sentences. Estimating the adjectives’ opinion orientation based 
upon seed word lists was exploited by [4], [5]. Likewise, [6] identified the opinion of review sentences 
through the average opinion tendency of word phrases in opinion text with pointwise mutual information 
(PMI) and latent semantic analysis (LSA) stood on the association with positive or negative seed word lists. 
A thesaurus was created by [7] to build the sentiment polarity specified system to opinion text. The syntactic 
interrelation between sentiment and target word with sentiment seed word list in [8] was applied to obtain 
sentiments and target words simultaneously. According [9] extracted aspect and detected the opinion 
orientation by using mutual information measurement and Naive Bayes. 

Opinion mining formed on (ii) was as well attracted at a recent time. In this approach, a set of 
techniques for classification are applied for categorizing opinion documents, detecting sentiment with a 
supervised learning mechanism on the two labeled datasets: training and testing set or unsupervised learning 
mechanism on the unlabeled data. In [10] used hidden markov model (HMM) for learning patterns to 
extricate aspects and opinion phrases in web opinion mining. In [11] extracted opinion target words in the 
document by using the conditional random field (CRF) technique. In [12] also applied the same CRF with 
two integrated modifications for obtaining and classifying aspects and opinions. Support vector machine 
(SVM) technique was used in [13] to extract opinion target words and classify the opinion polarity 
corresponding to each aspect. The three deep learning techniques recursive neural network, matrix-vector 
recursive neural network, and recursive neural tensor network were combined in one graded groundwork in 
[14] for extracting aspects and sentiment. A recursive neural network expansion in [15] with the received 
syntactical material from diagrams of dependency and component in opinion sentence was applied to classify 
the opinion of each aspect in text. In [16] used convolutional neural network (CNN) and the two variant 
models of CNN: parameterized filters for CNN and parameterized gated CNN to effectively achieve aspect- 
specific features for enhancing the accuracy of their model. In [17] used a recurrent neural network (RNN) 
variant to identify the opinion depended on target and context word in the sentence. Also with this context- 
based approach, evaluated the valuable level of context word aimed at their aspect-based sentiment analysis 
based on a neural attention architecture [18]. A method of continuous word embedding was used by [19] with 
the extraneural layer in their model to extract features and sentiments simultaneously. In [20] made use of the 
combination of RNN and CRF to identify aspects and sentiments. In [21], [22] applied effective attention 
techniques for aspect-based sentiment analysis (ABSA) but [21] focused on the content while [22] used a 
model of long short-term memory (LSTM) that based attention technique for putting mind to capture 
semantic and syntactic material to get the aspect’s necessary information for sentiment analysis. In [23] also 
used LSTM accompanying attention mechanism for taking the vital portions related to aspects in the sentence 
help to classify the opinion in ABSA. In [24] found the sentiment of each aspect in opinion text employing 
multiple attention control mechanisms on memory in a recurrent attention network. Our previous approaches 
in [25], [26] recommended the two models with the integration of CRF and RNN variants: gated recurrent 
unit (GRU) and IndyLSTM. We made the best use of the two-way mechanism for BiGRU and Bi-IndyLSTM 
to classify aspects proficiently. The incorporation of convolutional layers in CNN and the independent 
operation structure of gates was used in [27] and got good results in ABSA. 

Within the scope of this research, we offer to utilize the deep learning approach with the combined 
construction of CNN and bidirectional long-short term memory (BiLSTM) for multitasking ABSA model to 
extract and classify opinion target, aspect, entity and sentiment polarity concurrently. We decide to develop 
this model because of the subsequent details: CNN is capable of extracting local features based on 
convolutional kernel architecture with different sizes. Besides, LSTM has the capability to maintain memory 
in the long-term cycle and resolve the gradient vanishing and exploding concern to well handle long 
sequences. Through experimentation, the results obtained from our proposed ABSA model exhibit superior 
performance compared to previous models on the same dataset. Specifically, our approach has reached 
enhanced exactness on (i) two measures Fl and accuracy in terms of opinion target extraction task and 
(ii) four measures F1, accuracy, precision and recall on the aspect, category and sentiment classification task. 

We set up the layout of this paper for the rest in such a way. Section 2 illustrates our proposed 
approach with the two models: deep CNN_IOB2 model for extracting opinion target and MABSA model for 
classifying the remaining aspect, entity and sentiment polarity components. Our experimentations and 
argument about obtained products are shown in section 3. In summary, section 4 emphasizes our assumption 
and potential study tendency. 
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2. METHOD 

We propose two combined models: a Deep CNN-IOB2 and a MABSA model supported to main 
subtasks in ABSA. Our use of the CNN [28], [29] in combination with BiLSTM aims to take advantage of 
both CNN’s good local feature extraction proficiency and BiLSTM’s ability to learn long-distance 
representations in both dimensions for better mastering the sentence context. LSTM [17] is a better version of 
RNN with the ability to learn long-term dependencies without having problems with vanishing and exploding 
gradients [30] like RNN. Additionally, using softmax at the last layer in the model acts as a classifier to 
identify components concurrently. This incorporated method works for rising model correctness in 
multitasking accomplishment: opinion target, category, aspect and sentiment classification. 


2.1. Convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) 

Convolutional neural network (CNN) [29] has recently been exploited in natural language 
processing and triumphed notable successes. In the process of operating with sequences, convolutions take 
advantage of the competence to mine local features round each token. Each review with length 
L (n4,n2,...,nz,) after passing through GloVe word embedding layer and the POS word morphological 
determination layer is fed to the CNN to extract high-level features 

One of the two variants of RNN is LSTM [17] which is suitable for sequential data and solves 
problems of vanishing and exploding gradients encountered by RNN. With a mechanism of using three gates 
ft, it, 0, respectively forget, input and output gate, LSTM can filter and decide which information should be 
forgotten, updated, and outputted. In this paper, we utilized a bidirectional LSTM network to make use of the 
ability to learn far representations from the current token in both directions. This capability of BiLSTM helps 
to understand sentence context more clearly and removes information vagueness compared to the 
unidirectional nature of LSTM. 


2.2. Data preprocessing 

Data cleaning: some data cleaning operations are performed before word embedding. We detach 
emojis, uncommon characters, numbers and null sentences; change text to lower-case letter; find and 
substitute lingo or condensation with replacement (i.e. ‘vs.’ -> ‘versus’); lemmatize words in the sentence 
into their base form (i.e. ‘caring’, ‘cared’ -> ‘care’); word embedding: we use pre-trained word vectors from 
GloVe [31], an unsupervised learning technology, for learning word representation; part-of-speech (POS) 
tags: to identify the parts of speech of words in the input text, we use the stanford POS tagger; IOB2 tags: to 
obtain the opinion target word, we use IOB2 tags at the last layer of our Deep CNN-IOB2 model. Labeling 
B/I/O to each token in input sentence must comply with subsequent instructions: (i) the first word in the 
phrase is labeled “B-‘ or ‘O’, not ‘I-‘; (ii) valid label formats must be ‘O B-label’ or ‘B I-label’, not ‘O I- 
label’; (iii) each opinion target that can be a word or a phrase will have the first label ‘B-‘ not ‘I-‘. 


2.3. Proposed multitask aspect-based sentiment analysis model 
2.3.1. Deep CNN-IOB2 (DCI) model 

The GloVe word embedding layer was applied at the first position of our DCI model for encoding 
the semantically and syntactically related properties of a word in an input sentence with a 300-dimension 
feature vector. By concatenating with the POS tagger vector in the next layer to determine the parts of speech 
of the word, we get the 333-dimension feature vector and feed for the next CNN layer. In this paper, we used 
the three-layer CNN to extract high-level abstract features. The IOB2 architecture in the final layer position 
was superimposed on CNN layers in Figure | to assign B, I, or O to each token for the purpose of achieving 
the task of opinion target extraction. The labeling process pursues the following rules: B and I are labeled for 
the opinion target word at the beginning and inner location of the phrase, correspondingly, and O is labeled 
for the outer word. 


2.3.2. Multitask aspect-based sentiment analysis (MABSA) model 

Words in input review after being cleaned in the data preprocessing step will be provided to the 
word embedding and the POS tagger layer to encode into the 333-dimension feature vectors. In this model 
that can be shown in Figure 2, we still apply the three-layer deep CNN in the next layer for extracting local 
features around the token under consideration like our DCI model. A BiLSTM used in the succeeding layer 
supports the ability to learn information from both previous and subsequent directions at long steps away 
from the current token. Applying the average pooling layer on top of the BiLSTM aims to decrease the 
dimensionality of word vectors. In the last layer, we exploit a dense with softmax function for computing 
probability distribution over the possible categories to extract what we are looking for: the category, aspect, 
and sentiment simultaneously. Both of our models has reached outstanding correctness compared with earlier 
advanced researches. 
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Figure 1. DCI model for extracting opinion target 
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Figure 2. MABSA model for extracting and classifying aspect, category and sentiment 


3. RESULTS AND DISCUSSION 

We use the SemEval2016 Task 5 benchmark dataset as a corpus for training and evaluation. 
Our DCI and MABSA models have been experimentalized on the restaurant domain that can be shown in 
Table 1 and provided significant performance. Fl and Accuracy are the two measures used in our 
experiments to evaluate the performance of proposed models compared with previous models also 
experimented on the restaurant domain of the SemEval 2016. Each concatenation vector of 300-dimensional 
GloVe feature vector and POS tagger vector of 33 dimensions corresponding to each word in the input 
sentence is fed to the next CNN layer. 


Table 1. Restaurant domain 


Restaurant domain Sentence Opinion target 
Training 2000 1743 
Testing 676 622 


CNN will perform the task of extracting high-level features and fed to the next layer of BiLSTM. 
We use the rmsprop optimizer with a learning rate of 0.001, dropout 0.25, and a batch size of 128 in both 
DCI and MABSA models. Tables 2 and 3 describe our experimental results on the two proposed models. Our 
proposed DCI model for extracting opinion target words or phrases of input reviews is evaluated through 
comparison with the following baseline models also experimented on the same benchmark dataset_the 
SemEval 2016, restaurant domain. Besides, our MABSA model of classifying the remaining subtasks such as 
category, aspect, and sentiment polarity also achieved remarkable results. 
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Table 2. Compare the experimental results of our DCI model with baseline models in F1_score and 
Accuracy_score on the restaurant domain of the SemEval 2016 dataset 


Model Opinion target extraction 
NLANGP(U) [32] FI_score Accuracy_score 

CRF [33] 72.34 - 
AUEB [34] 66.54 
MIN [35] 70.44 
DE-CNN [36] 73.44 
THA&STN [37] 74.37 
BiDTreeCRF [38] 73.61 
Our DCI model 74.49 


As can be seen obviously from Table 2, with the task of extracting opinion target words, the DCI 
model has surpassed the former researches and reached superior performance on both Fl and Accuracy. 

—  Fl_score: Our DCI model achieves 27.62%, 33.42%, 29.52%, 26.52%, 25.59%, 26.35%, 25.47% higher 
accuracy than previous NUANGP(U), CRF, AUEB, MIN, DE-CNN, THA&STN, BiDTreeCRF models 
respectively on the same Restaurant domain. 

—  Accuracy_score: Our model achieves a pretty high result of 99.93%. 

The graph from Figure 3 depicts our opinion target extraction model’s validation Fl, accuracy 
measure, and loss. Viewing the loss chart from Figure 3, we see that after a drastic decrease for the first five 
epochs and a slower decline over the next 35 epochs, the loss has been on a stable downward trend since the 
40th epoch and reached a steady-state between 100th and 120th epoch. From Figure 4, it is clear that our 
deep CNN-IOB2 model attains superior results in the two measures Fl and accuracy compared to previous 
methods. 
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Figure 3. Fl, accuracy score and training loss of our deep CNN-IOB2 model 
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Figure 4. Fl, accuracy of previous models and our deep CNN-IOB2 model in opinion target extraction 


Similar to the DCI model above for extracting opinion target words, the MABSA model for 
classifying aspect, category and sentiment has also achieved outstanding results. Specifically, in Table 3, the 
model’s results for category and sentiment classification are really high in the range of 96.58% to nearly 98% 
on accuracy, precision, recall, and Fl. The experiment results of our model for aspect extraction are also 
relatively high, reaching over 92% on all measures. 
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Table 3. Experimental results of our MABSA model in accuracy, precision, recall, Fl_score on the domain of 


restaurant 
Our MABSA model Category classification Aspect extraction Sentiment polarity classification 
Accuracy 97.83 92.39 96.58 
Precision 97.97 92.52 97.19 
Recall 97.67 92.24 96.58 
Fl 97.82 92.38 96.88 


Figure 5 depicts the performance evaluation of the model in terms of two measures F1 and accuracy. 
In addition, the last three charts in the group of descriptive charts describe the model’s loss. In the category 
loss chart, after a drastic drop in the first 50 epochs and a stable downward trend from the 50th epoch to the 
150th epoch, the loss tends to be steady from the 150th epoch to the last 300th epoch. A similar downtrend is 
also shown in the aspect loss chart with a sharp decrease in the initial 20 epochs. The downturn will be 
lighter between the 20th epoch and 120th epoch. From the 120th epoch on, the chart shows a very trivial 
decrease until the 200th epoch and gradually stabilizes in the last interval. The ultimate chart for polarity loss 
has a slight difference from the prior two with a deep drop from the 50th epoch to 125th epoch after the first 
50 epochs with a negligible decrease. During the period from the 125th to the 175th epoch, the chart tends to 
decrease slightly, repeat as at the beginning and stabilizes after the 200th epoch. As is illustrated by the chart 
in Figure 6, our MABSA model for the remaining tasks achieves high performance in the four measures of 
precision, recall, Fl, and accuracy. 
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Figure 5. Accuracy, F1 score and loss of our MABSA model 
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Figure 6. Precision, recall, Fl and accuracy score of our MABSA model 


4. CONCLUSION 

Within the scope of this study, we have suggested an approach of deep learning for multitasking 
ABSA that completes opinion target, category, aspect, and sentiment polarity classification tasks 
simultaneously with two models DCI and MABSA. On both models, we used a deep 3-layer CNN design 
with feature vectors derived from the previous GloVe combined with the POS tagger layer to enable 
high-level abstraction feature extraction. For the DCI model that supports the opinion target, we implemented 
an IOB2 layer at the end of the model right on top of the CNN layer to label each word with one of three 
B/I/O labels. Deployment of a BiLSTM layer on top of the CNN in our second model based on the advantage 
of long-distance learning in both directions integrated with a fully connected layer and a softmax at the top 
layer of the model contributed to making a classifier for aspect, category, and sentiment. Our proposed 
approach has significantly succeeded on the same experiment domain of the benchmark dataset over the 
preceding baseline models. In upcoming research, we plan to deploy our models on a multidomain dataset to 
increase the utility and friendliness of the models. 
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